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Description 

Field of the Invention 

5 [0001] The present invention relates to handheld devices and in particular, but not exclusively, to handheld devices 
such as mobile telephones or the like. 

Background to the Invention 

10 [0002] Wireless cellular telecommunication networks are known. The area covered by a telecommunications network 
is divided up into a number of cells. Each cell has a base transceiver station associated with it. The base transceiver 
station is arranged to send signals to and receive signals from mobile stations in the same cell as the respective base 
station. The signals sent between the mobile stations and the base station permit voice and/or data communications 
to take place. 

15 [0003] In order to prevent the unauthorised use of mobile stations by other people, password protection is used. In 
order to activate the keyboard and/or allow the user of a mobile phone to make a call, the user has to enter a password. 
That password consists of a number. 

[0004] One problem with the use of numbers is that it can be difficult for the user to remember the password. This 
is both inconvenient and annoying to the user. 

20 [0005] Alternative methods of preventing unauthorised use of a mobile phone have been proposed which rely on 
finger print recognition. The phone can be set up so that it can only be activated if it correctly identifies the user's 
fingerprint. However, if fingerprint data is stolen, for example, from a central store, it is not possible for the user to alter 
his fingerprint. Accordingly, the unauthorised use of the mobile phone may not be preventable, 
[0006] One f urth er problem is that the password used by the user may be simple for an unauthorised person to guess 

25 and once the correct password has been guessed, the mobile telephone can be used without the consent of its owner. 
People often use their birthdays or the name of a pet. 

[0007] The optimal settings for a mobile telephone, such as its volume level and the like may depend on whether 
the mobile telephone is stationery, the user is walking, cycling or in a car. Currently, in order to achieve the optimal 
settings, the user has to change the settings as the environmental changes. This is inconvenient and often the user 
30 forgets to do this. The user may forget to change the settings before the user receives a call and the user then has to 
either alter the settings during the call or conduct the call with the less than optimal settings. 

Summary of the Invention 

35 [0008] It is an aim of embodiments of the present invention to address one or more of these problems. 

[0009] According to one aspect of the present invention, there is provided a mobile communications device compris- 
ing means for determining the orientation of said device; and trained signal processing means for processing an output 
of the determining means. 

[0010] According to a second aspect of the present invention, there is provided a mobile communications device 
40 comprising means for detecting movement of the device and trained signal processing means for processing an output 
of the detecting means. 

According to a third aspect of the present invention, there is provided a hand held device comprising means for deter- 
mining at least one of orientation and movement of the device; and processing means for recognising at least one 
predetermined orientation and/or movement of said device. 
45 [001 1] The following preferred features of embodiments of the present invention can be used in conjunction with one 
or more of the above aspects. 

[0012] Preferably, the trained signal processing means are arranged to recognise at least one predetermined move- 
ment. The trained signal processing means may be arranged to determine the orientation of the device. The trained 
signal processing means may be arranged to recognise at least one predetermined position. 
50 [0013] The processing means may be arranged to recognise a sequence of movements. The trained signal process- 
ing means may be arranged to determine the current mode of operation of the device. For example, depending on the 
orientation and/or movement of the device, it is possible to determine if the device is in a meeting mode, a games 
mode or the like. 

[0014] Preferably, control means are provided to control the device to provide an associated function when a pre- 
ss determined orientation and/or movement is recognised. For example, the device may be usable only if a predetermined 
orientation and/or movement is recognised. In other words, passwords can be implemented using a predetermined 
orientation and/or movement. The associated function may be a function of the user interface of the device. For ex- 
ample, where the device is a mobile telephone, a predetermined orientation and/or movement may be used to answer 
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a call, close connection, switch off an alarm or the like. 

[0015] The means for detecting the movement of the device may comprise at least one accelerometer. The means 
for determining the orientation of the device may comprise at least one accelerometer. In preferred embodiments of 
the present invention, at least one accelerometer is arranged to provide movement and orientation information. Pref- 
5 erably, three accelerometers are provided which are arranged to provide information in respect of three orthogonal 
directions. 

[0016] The trained signal processing means may comprise at least one neural network. The at least one neural 
network may be used for recognising the orientation of the device. The at least one neurai network may use a self 
organising mapping technique. 

10 [0017] The trained signal processing means may comprise at least one pattern recognition system, for example a 
hidden Markov model. 

[0018] The predetermined orientation and/or movement of the device may be pre-programmed or alternatively may 
be programmed into the device by the user or selected by the user. 

[0019] Preferably, the device is a mobile telephone. It should be appreciated that in alternative embodiments of the 
15 invention, the device can be any other suitable device, such as a control device for a game or the like. 

[0020] According to a fourth aspect of the present invention, there is provided a method for controlling a device 
comprising the steps of determining the orientation and/or movement of the device; using a trained signal processing 
method to process the result of the determining step; and controlling the device in response to the determined orien- 
tation and/or movement of the device. 

20 

Brief Description of the Drawings 

[0021] For a better understanding of the present invention and as to how the same may be carried into effect, refer- 
ence will now be made by way of example to the accompanying drawings in which:- 

25 

Figure 1 shows a cellular telecommunications network with which embodiments of the present invention can be 
used; 

Figure 2 shows a block diagram of a mobile telephone embodying the present invention; 
Figure 3 shows a schematic view of an accelerometer used in the arrangement of Figure 2; 
30 Figure 4 shows an example of signals which are output by the accelerometer in Figure 3; and 

Figure 5 illustrates the function provided by the windowing circuitry of Figure 2. 

Detailed Description of Embodiments of the Invention 

35 [0022] Reference will first be made to Figure 1 which shows a wireless cellular telecommunications network 2. The 
area covered by the network 2 is divided up into a plurality of cells 4. Each of the cells 4 has associated with it a base 
transceiver station -6. Each base transceiver station 6 is arranged to communicate with mobile stations 8 in the cell 
associated with that base transceiver station 6. 

[0023] The network shown in Figure 1 may use any suitable method of communication for communicating with the 

40 mobile stations. In particular, one or more of the following methods may be used: Time Division Multiple Access, Fre- 
quency Division Multiple Access, Spread Spectrum Methods such as Code Division Multiple Access or other suitable 
methods. In some embodiments of the present invention, hybrids of two or more of these access methods may be used. 
[0024] Reference will now be made to Figure 2 which shows an embodiment of the present invention, arranged to 
be incorporated in a mobile telephone. The elements shown in Figure 2 are arranged to provide information as to the 

45 orientation of the mobile telephone and its movement. 

[0025] The arrangement shown in Figure 2 comprises three accelerometers 10,12 and 1 4. The first accelerometer 
1 0 is arranged to detect motion in the X direction. The second accelerometer 12 is arranged to detect motion in the Y 
direction. The third accelerometer 14 is arranged to detect motion in the Z direction. In other words, the three accel- 
erometers 10, 12 and 14 are each arranged to detect motion in mutually perpendicular directions. 

so [0026] Reference is made to Figure 3, which shows a schematic example of one accelerometer which can be used 
with embodiments of the invention. The accelerometer comprises a mass 100. The accelerometer shown in Figure 3 
is arranged to detect motion in the directions of arrow A. When there is no motion in the directions of arrow A and there 
are no external forces on the mass in the directions of arrow A, the mass will be in its rest position shown in Figure 3. 
However, when there is movement in the direction of arrow A, the mass will move about the rest position indicated by 

55 dotted line B. It should be appreciated that if gravity provides a force in one of the directions of arrow A, the rest position 
will be above or below line B, depending on the orientation of the sensor. 

[0027] Reference is made to Figure 4 which shows an output of the accelerometer shown in Figure 3. The sensor 
is initially oriented in such a way that there is a force on the mass 1 00 in one of the directions of arrow A due to gravity. 
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The part of the graph 102 represents the condition where the sensor is in one orientation such that it is affected by 
gravity in one of the directions of arrow A and there is movement in at least one of the directions of arrow A. The part 
of the graph referenced 1 04 shows the output of the accelerometer when the sensor is in the opposite orientation so 
that gravity still affects the sensor but in the other direction of arrow A. This can be because the phone has been turned 
5 upside down. There is also movement in at least one of the directions of arrow A. Thus, graph 102 can be said to be 
when gravity acts in the direction of arrow C whilst part of the graph referenced 1 04 is where gravity acts in the direction 
of arrow D. 

[0028] As can be seen in Figure 4, the part of the graph referenced 1 02 provides a signal based about a centre point 
represented by line E whilst the part of the graph referenced 104 provides the signal centred about the line F. Lines E 
10 and F provide information about the position of the phone whilst the signals themselves provide information about the 
amount of movement. It should be appreciated that depending on the orientation of the phone, the signals can be 
centred about different lines which will be between lines E and F. This is because lines E and F represent the two 
extreme positions. 

[0029] Each of the sensors shown in Figure 2 provides a signal of the type shown in Figure 4 but for the three 

15 orthogonal directions X, Y and 2. The signals thus provide information about the position of the phone in the X, Y and 
2 directions as well as information about movement of the phone in the X, Y and 2 directions. 
[0030] In preferred embodiments of the present invention, accelerometers such as the ADXL 202 made by Analog 
devices may be used. Any other type of accelerometer can be used which may use different techniques in order to 
sense gravity to provide position information and acceleration to provide movement information. 

20 [0031] The output of the accelerometers 1 0 to 1 4 are input to an analog to digital converter 1 6 which converts the 
analog outputs of the accelerometers into digital signals. It should be appreciated that in alternative embodiments of 
the present invention, the accelerometers may provide digital outputs, in which case the analogue to digital converter 
can be omitted. In still further embodiments of the present invention, it may be possible to process the analog outputs 
of the accelerometer in their analog form and thus the need for the analog to digital converter can be avoided. 

25 [0032] The operation of the accelerometers 1 0 to 1 2 will depend on the temperature of the environment in which the 
mobile telephone is located. The temperature sensor 24 thus measures the temperature inside the mobile station. The 
temperature sensor 24 may take any suitable format. The output of the temperature sensor 24 is connected to an 
analog to digital converter 26. The output of the temperature sensor 24 is an analog signal which is converted by the 
analog to digital converter 26 to a digital signal. 

30 [0033] The digital signal representing the temperature of the mobile phone is input to a calibration arrangement 22. 
This calibration unit 22 provides a correction output which is used to correct the digital signals representing the output 
of the accelerometers 10 to 14. In this way, the influence of temperature on the accelerometers can be disregarded 
so that a more accurate evaluation of the movement and position of the mobile telephone can be made. 
[0034] The calibration unit 22 can take any suitable format. For example, the calibration unit can take the form of a 

35 look up table, with the digital output of the analog to digital converter 26 providing an address to the look up table. 
Alternatively, the calibration unit may perform an algorithm which calculates the compensation required based on the 
output of the analog to digital converter 26. The calibration unit 22 can of course use a combination of an algorithm 
and a look up table. 

[0035] The analog to digital converter 1 6 connected to the accelerometers 1 0 to 1 4 provides two identical outputs. 

40 Each output contains a digital representation of each of the three outputs of the three accelerometers 1 0 to 14. 

[0036] One of the outputs is connected to a low pass filtering arrangement 18. The low pass filter is arranged to 
remove frequency components above a given cut of frequency. For example, with the signal shown in Figure 4, the 
low pass filter 18 will remove the part of the signal above line E for the part of the signal referenced 102 and will remove 
the part of the signal above line F for the part of the signal referenced 104. The function of the low pass filter 18 is 

45 effectively to allow the middle positions of each portion of the output from each sensor to be determined as this provides 
information as to the position of the mobile telephone. Alternatively, a DC estimator can be used to provide orientation 
information. 

[0037] In the example shown in Figure 4, lines E and F are straight lines. However, it should be appreciated that 
depending on the motion of the mobile telephone, the line about which the signals are centred may be curved or any 
50 other possible shape. 

[0038] The output of the low pass filter 1 8 is input to a position recognition system 30 in a post processing layer 28. 
The position recognition system 30 includes an artificial neural network 32. The neural network 32 is trained to identify 
the position of the mobile telephone based on the filtered outputs which it receives from the low pass filter 1 8 for each 
of the three sensors. For example, one combination of values from the three accelerometers 10 to 14 will be provided 
55 when the mobile phone is lying flat on a table with its display uppermost. A different combination of outputs from the 
three accelerometers will be provided when the mobile phone is in a vertical position and so on. 
[0039] Any suitable type of artificial neural network ANN or pattern recognition unit can be used. By way of example, 
the artificial neural network 32 that can be used is the self organising map proposed by Kohonen. Another example of 
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an ANN is a multi layer perception neural network. This ANN is used for pattern recognition arid feature extraction. In 
embodiments of the present invention, this ANN is able to extract information relating to the position of the mobile phone. 
[0040] A self organising map SOM is a neural network which forms spatially organised feature maps from an N 
dimensional input signal in an unsupervised manner. In the embodiment of the present invention, N is equal to 3 with 
information being provided by each of the three sensors. The method used by the self organising map is similar to 
human sensory input mapping in the brain, which is then organised topographically. During training, weight sectors Wjj 
(n) of the network are shifted closer to the input sectors X by 



10 



Wj,(n+1 )=w ji (n)+k(n)n j . 1(x) (n)(x-w j1 (n)). 
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where Wj,(n+1 ) is the updated weight vector, k(n) the learning rate and nj. 1(x) (n) the neighbourhood function. The learning 
rate and the neighbourhood function are changed as the training proceeds. After self-organization the hand labelled 
training data presented to the SOM again and labels are suggested for the winning neurons with their immediate 
neighbours. All suggestions are collected for each neuron, after which majority voting is performed for final labelling. 
[0041 ] The three dimensional sensor data is normalized priorto the ANN. that is the mean and variance arecalculated 
and used for computing the normalization. A normalization block 31 can be implemented into the position recognition 
system 30. The normalisation block 31 receives the output of the low pass filter 1 8 and the output of the normalisation 
block is input to the ANN. 

[0042] The artificial neural network unit 32 is arranged to provide two functions, feature extraction and classification. 
Feature extraction allows the neural network to process the information which it has received. The output is then 
classified so that it can be determined what has occurred. In one embodiment of the present invention, the neural 
network 32 is trained to identify the following positions: 



25 



30 



the mobile phone is in an orientation with the display upwards, 
the mobile phone is in an orientation with the display downwards, 
the mobile telephone has atypical in the pocket orientation, 
the mobile phone is being held to the user's left ear 
the mobile phone is being held to the user's right ear, 
the mobile phone is in an orientation on its left side, 
the mobile phone is in an orientation on its right side, 
the mobile phone is standing, and 
the mobile phone is upside down. 



35 



40 



45 



50 



[0043] It should of course be appreciated that embodiments of the present invention may be used to identify a larger 
range of orientations of the mobile station or indeed any orientation of the mobile station. 

[0044] In one embodiment of the present invention, the self organising map used is a two dimensional hexagonal 
grid arranged to be seven neurons by seven neurons. The parameters of the self organisation map are shown in table 
1 below. 

Table 1 



Parameter 


Value/Argument 


Map lattice 


Hexagonal 


Initialization type 


Linear 


Size of the map 


7x7 


Initial and final learning radiuses 


[3 1] 


Initial learning rate 


0.03 


Learning rate 


Linear 


Neighbourhood function 


epanechnikov 



55 



The signals from the accelerometers are sampled. In one embodiment they are sampled with a frequency of 90Hz. 
The input to the low pass filter 1 8 starts a sequence of vectors of the following form: 
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where X n , Y n and Z n are acceleration signals in X, Y and Z directions at a discrete time IM. In the low pass filter 18, 
each vector component is filtered and normalised separately. A low pass filtering is carried out using, in one embodiment 
of the present invention, a fourth order Butte rworth filter of the type IIR. A3-dBcut off frequency of 2.5Hz was used in 
one preferred embodiment of the present invention. The variance of each component is normalised to 1 and the mean 
to zero. The processed components of each vector are recombined to a sequence of 3 dimensional feature vectors P' n . 
[0045] The feature vector P' n at time N is input to a 2 dimensional feature map of the artificial neural network 32. 
This produces an index l n representing thetraining data cluster in question. The resulting sequences of self organising 
map indices l n is interpreted by the classifier which assigns a label to each index according to the training data. Each 
label suggests a specific gesture or position of the mobile telephone, for example display up etc. Finally, a majority 
voting is formed among the label sequence to recognise the gesture. 

[0046] The position recognition system 30 provides two outputs. The first output provides position information on the 
phone which can be used as required. The second output is to a dynamic acceleration recognition system 35 which 
will be discussed in more detail hereinafter. 

[0047] One example of an artificial neural network is a multi layer perception network (MLP). This has three input 
layer neurons, ten hidden layer neurons and six output layer neurons. The network can produce position data as 
number sequences of six numbers. Each number can represent a different one of the positions. For example: 

• Display up: 001000 

• Display down: 100000 

• Device standing: 0001 00 

• Bottom up: 010000 

• Sideways to the left: 000001 

• Sideways to the right: 00001 0 

[0048] The output of the analog to digital converter 1 6 connected to the outputs of the accelerometers 1 0 to 1 4 is 
also connected to a high pass filter or AC estimator 20. The high pass filter is arranged to remove the effects of, for 
example, gravity from the signal. In the case of the signal shown in Figure 4, the high pass filter 20 is arranged so that 
the first part of the signal 102 and the second part of the signal 104 are centred about a common line having a zero 
or similar value. If the signal is such that the centre line curves, again, the signal will be filtered such that the signal is 
centred about a straight horizontal line at the same zero or other reference point. This is because the signals are being 
used to identify information relating to the movement of the mobile telephone and not its position. 
[0049] The output of the high pass filter 20 is input to a windowing unit 32. The windowing unit 32 receives three 
signals, an example of which is shown in Figure 5. One signal represents the output of the X accelerometer 10, the 
second input represents the output of the Y accelerometer 1 2 and the third input represents the output of the Z accel- 
erometer 14. Each of the signals is centred about a zero level. The windowing unit 32 defines a window. The signals 
within a window are processed together in order to obtain the required information. As can be seen from Figure 5, the 
first window shown, window A overlaps slightly with the next window, window B. It should be appreciated that all of the 
signals output by the high pass filter will be divided into windows for processing. The filtering may use any form of DSP 
filters. 

[0050] The output of the windowing unit 32 is connected to the dynamic acceleration recognition system 35. One 
output of the windowing unit 32 is input to a first vector quantization unit 34. The vector quantization unit 34 quantizes 
the signals received in a given window into a plurality of vectors each having an X, an Y and an Z value. The values 
are derived from the respective X, Y and Z signals. The quantized signals are output to a second pattern recognition 
system 36. This system is, in preferred embodiments of the present invention, a hidden Markov Model (HMM). 
[0051] The windowing unit 32 also provides an output to a fast Fourier transform unit 40. This unit converts the 
received signals from the time to the frequency domain. The output of the fast Fourier transform unit 40 is input to a 
frequency peak estimate unit 42. The frequency peak estimate unit 42 identifies, for each of the X, Y and Z signals the 
highest frequency. The identified highest frequency for each of the X, Y and Z signals is output to a second vector 
quantization unit 38. This quantizes the output of the frequency peak estimate unit 42 and outputs it to the HMM 36. 
[0052] The fast Fourier transform unit 40 also outputs the X, Y and Z signals in the frequency domain to the second 
vector quantization unit 38 which also quantizes the three X, Y and Z signals and outputs them to the neural network 36. 
[0053] Based on the information which the neural network receives from the first and second vector quantization 
units 38 and from the position recognition system 30, the neural network 36 is able to identify how the mobile station 
is moving. 

[0054] The hidden Markov model 36 is a stochastic process with an underlying process of transitions between hidden 
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states of the system and a process of emitting observable outputs. When the outputs are discrete symbols, this is 
referred to as discrete HMM. The state transitions form a first order discrete Markov process with a transition probability 
distribution A and in an initial state distribution n. The observable process of emitting symbols can be presented as an 
observation symbol distribution B. Thus each HMM X can be represented as a triplet-X=(A, B, %). In the embodiment 
5 of the present invention, Baum-Welch and Viterbi algorithms have been used for the training and recognition tasks. 
The loq-Viterbi form of the Viterbi algorithm has been used as this is economic with computational processing. Before 
training, the initialisation of the HMM parameters were carried out as follows: 

1 . The initial state probability for the first state was set to 1 . 
10 2. The transition probability distribution for each state was set uniformerly distributed; and 

3. Any topologically allowable state transition from individual states was given probabilities of the form 1 /(amount 
of allowable state - transitions from state). 

[0055] A HMM with a left right topology is used for modelling time series whose properties change sequentially over 
15 time. This model was used in one embodiment of the present invention. In one implementation, left-right HMM models 
with 7 states were used. It has been found that the number of states was not particularly critical and that there was 
little difference between choosing five or four states on the recognition ability of the system. Any other type of HMM 
may be used in alternative embodiments of the invention. 

[0056] A collection of the three dimensional acceleration data is performed with a 1 00Hz sampling rate. As described 
20 previously, each signal from the X, Y and Z accelerometers 1 2 to 1 4 are separately filtered with the high pass filter 20. 
In particular, a fourth order low pass Butterworth filter with a 3dB cut off frequency of 4Hz is used. The signal was 
decimated at 1/6 times the original rate. 

[0057] The HMM 36 detects the usable signal, this is also referred to as gesture segmentation. For individual ges- 
tures, normalisation of each component to a zero mean and unit variants is performed. 

25 [0058] The vector quantization units 34 effectively act as a code book. Discrete code book indices correspond to the 
observable symbols and are input to the HMM both in the training and test phase. The indices are computed by vector 
quantization in the first vector quantization unit 34 of the 3D input vectors of the acceleration signals from the accel- 
erometers. The code book used by the vector quantization units is constructed by uniformerly quantizing the 3D feature 
space. The uniform quantization is advantageous because the acceleration trajectory in the feature space can pass 

30 through any point with equal probability within the region defined by the application. The maximum values of each 
acceleration component was searched in the measured training data in order to define the parallelogram of activity in 
the feature space. In one preferred embodiment of the present invention, the code book was of 512 3 dimensional 
codewords. The larger the size of the code book, the more accurate the recognition capability of the recognition system. 
Accordingly, the degree of recognition which is required will determine the size of the code book. 

35 [0059] The vector quantization carried out by both of the vector quantization units is performed in a conventional 
way by selecting the code book entry containing the closest code word to the input vector in the Euclidean sense. 
[0060] The HMM 36 can thus be trained to recognise certain gestures or movements of the phone. This will rely on 
a combination of information relating to the position of the mobile telephone and the movement of the mobile phone. 
For example, using information on the position and/or movement of the mobile station, it is possible to identify the 

40 following: 

1 . The user is holding the phone in front of him and watching the display. This can be while the user is moving or 
stationary. 

2. The phone is lying on the table with its display downwards. 

45 3. The phone is against the ear of a user with the user either stationary or moving. 

4. The phone is in a pocket of a user with the user moving. 

5. A ringing phone, which is in a right hand side of a belt box with the display turned towards the user is taken into 
the right hand and the number of the caller is checked by looking at the display. After this, the phone is returned 
to the belt holder. 

50 6. A ringing phone, which is in a right hand side belt with display towards the user is taken into the right hand and 

the number of the caller is checked by the user viewing the display. After this, the user opens the line by pressing 
the corresponding button on the user interface and brings the phone up to his right ear. 

7. The phone call ends, the phone is taken from the ear and the line is shut down by activating a button on the 
interface. Finally, the phone is put back into the belt box. 
55 8. A ringing phone, which is on the desk with its display facing up is taken into the right hand and the number of 

the caller is checked by watching. After this, the phone is put back on the desk in the same position. 
9. A ringing phone, which is on the desk with its display facing upwards is taken into the right hand and the number 
of the caller is checked by watching. After this, the user opens the line by pressing the corresponding button on 
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the interface and moves the phone to the right ear. 

10. The phone call ends, the phone Is taken from the ear and the line Is shut down by activating a button on the 
display. Finally, the phone is put back on the desk with its display up. 

5 [0061 ] These are just some of the examples of different movements of the phone which can be identified by the ANN 
32and/ortheHMM36. 

[0062] The information provided by the post processing layer 28 including the neural network and HMM can be used 
in a number of different ways. For example, information as to the movement of the user can be ascertained. For 
example, it is possible to identify whether the user is moving or stationary. If the user is moving, it is possible to identify 
10 the type of movement and thus to identify if the user is walking, running, cycling, or in a car or the like. It is then possible 
to automatically select the settings of the user so as to optimise the quality with which the user is able to make and 
receive calls. For example, if the user is moving, it may be harder for the user to hear the ringing tone of the phone. 
Accordingly, the volume of the ringing tone may be automatically increased. 

[0063] In one embodiment of the invention, information on the type of movement of the user may be sent to the base 
15 station which uses that information for determining the base station with which the mobile station is to communicate, 
when handoff should occur or the like. 

[0064] In one embodiment of the present invention, the neural networks can be trained to recognise certain gestures 
and to control the phone accordingly. For example, instead of a number which is input to the user interface to provide 
password protection, a gesture can be used. For example, the gesture may consist of the user signing his name in the 
20 air or against a surface with the mobile phone. Alternatively, the gesture may comprise a set of movements. 

[0065] Using the position and movement neural networks, the HMM is able to identify when the user performs the 
password gesture and to allow the phone to be used when that gesture has been detected. 

[0066] It should be appreciated that the neural network is able to identify the gesture from the movement of the phone 
and the position of the phone over a period of time. Typically, but not always, the gesture will span several windows. 
25 [0067] Gestures can also be used to achieve certain functions. For example, a series of gestures may be used to 
automatically answer the phone, for example, a phone could be shaken in a particular way for a predetermined number 
of times. 

[0068] It is in fact possible to allow the user to control virtually any aspect of the operation of the phone which can 
be usually controlled via the user interface using an associated gesture. The user can select his own gestures. He will 
30 need to train the neural network to recognise the gesture by repeating the gesture a number of times. The neural 
network will then learn to recognise that gesture and when that gesture is recognised causes the associated function 
to occur. 

[0069] Gesture based passwords can be used by a group of users to share some data. In this case, information to 
train the neural network to recognise the gesture may be transmitted to the mobile station or mobile stations from base 
35 stations. 

[0070] In an alternative application of this, the gesture password may be used to generate a game. For example, a 
user may transmit a ciphered file to the mobile phone of friend along with associated gesture password information. 
The friend may then try to get the right gesture in order to access the ciphered file. The mobile station can be pro- 
grammed to provide audio or visual indicators as to whether or not there is a match and how similar the gestures are. 

40 The mobile station can also be arranged to give the friend hints or the like. 

[0071] It should be appreciated that the gestures can be in two or three dimensions. The gestures can be in the air 
or against a surface. For example, the antenna of a phone could be used as a pen like member. 
[0072] In one embodiment of the present invention, the user may move the phone, for example as a pen to write 
messages in the air or against the surface. The information which is drawn or written by the user may be displayed on 

45 the display. In more sophisticated embodiments of the present invention, the letters which are written are recognised. 
In this way, a simple way to generate text messages is provided without having to use the keyboard which can be time 
consuming given that there are usually only a limited number of keys. 

[0073] The neural network can be trained so as to take into account the speed at which a gesture is formed and only 
to recognise that gesture if the timing is similar to that which has been prestored. However, in alternative embodiments 
so of the present invention, the neural network can be trained so as to ignore differences in timing. 

[0074] The password can be arranged to protect any suitable resource, such as access to the mobile telephone 
itself, access to the SIM card or access to a file. 

[0075] Embodiments of the present invention are applicable not only to mobile telephones but any other suitable 
hand held device. These hand held devices may be provided with acce I ero meters such as discussed hereinbefore. 
55 [0076] One example of an alternative device is one which includes an integrated alarm clock. That might be switched 
off by a shaking type of gesture. It should be appreciated that some mobile telephones may incorporate an alarm clock. 
[0077] Embodiments of the invention may be used with devices such as Tamogochi toys where the users has to 
provide a feedback in response to certain stimuli. The gestures can provide the feedback with different gestures being 
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used for different feedback responses. 

[0078] Gestures can be carried out in order to permit data to be logged in an automatic diary, reminder system or 
context aware applications. 

[0079] It should be appreciated that in certain embodiments of the present invention, the gestures used by the user 
can be changed either for variety or for security purposes. 

[0080] It should be appreciated that the neural networks provide a degree of tolerance for the matching of the per- 
formed gesture and the known gesture. In some embodiments of the present invention, the degree of tolerance which 
is allowable can be adjusted by the user. 

[0081 ] In preferred embodiments of the present invention, the mobile telephone or similar device will be sold with at 
least part trained neural networks. However, in alternative embodiments, the user may do some or all of the training. 
The training can be done at the device itself. The training can also be done as follows: 

- The device logs the sensor data during a gesture. 

The data is sent to a network server which carries out the required training. 

The server sends the ANN source code to the device. In this way, a new gesture can be recognised. This can be 
quicker than training in device itself. 

[0082] Gesture recognition can be used in order to control dialling functions of a mobile phone. It is also possible to 
use gesture recognition in order to play games with mobile telephones or any other such devices. The information from 
the accelerometers can be used to control the interface of the mobile phone in a number of ways. Games can be played 
by tilting the mobile station and detecting that movement and position. Dialling can be achieved by for example shaking. 
Certain positions of the phone can cause certain modes to incur and so on. 

[0083] In the embodiment described hereinbefore, discrete HMM is used. If continuous HMM is used, effective quan- 
tization would not be required. 

[0084] Embodiments of the present invention can be used to provide an automatic antitheft alarm system. For ex- 
ample, if the device is used with the incorrect gesture or is moved when it should be stationary, an alarm can be set. 
[0085] In another embodiment of the present invention, the device is a hand held device which acts as a control 
device, for example for a game or the like. The control device may be wired or wireless. The device can therefore act 
as a joystick or the like. 

[0086] It should be appreciated that any alternative neurai network to the artificial neural network or HMM network 
described in the embodiment of the present invention can be used in embodiments of the present invention. Multiple 
parallel recognition systems (ANNs and/or HMMs) can be used depending on the operating state of the device. 
[0087] In an alternative embodiment of the invention, the device or mobile telephone may have different modes. 
Those modes may for example be meeting mode, games mode or any other different modes. The gestures, orientations 
or movements which are detected in the different modes may result in different functions in different modes. Of course 
some gestures, movements or positions may have no significance in some modes. The neural networks may be trained 
to determine the mode from the position and/or movement of the device. For example, in a meeting mode, the mobile 
telephone may be stationary and lying on a horizontal surface with its display mode upwards. 

Claims 

1. A mobile communications device comprising: means for determining the orientation of said device; and trained 
signal processing means for processing an output of the determining means. 

2. A mobile communications device comprising: means for detecting movement of the device and trained signal 
processing means for processing an output of the detecting means. 

3. A device as claimed in claim 2, wherein said trained signal processing means are arranged to recognise at least 
one predetermined movement. 

4. A device as claimed in claim 2, wherein said trained signal processing means are arranged to determine the 
orientation of the device. 

5. A device as claimed in claim 1 , wherein said trained signal processing means are arranged to recognise at least 
one predetermined position. 

6. A hand held device comprising: 
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means for determining at least one of orientation and movement of the device; and 

processing means for recognising at least one predetermined orientation and/or movement of said device. 

7. A device as claimed in claim 6, wherein said processing means comprises trained signal processing means. 

8. A device as claimed in claim 2, wherein said processing means are arranged to recognise a sequence of move- 
ments. 

9. A device as claimed in claim 1 , wherein said trained signal processing means is arranged to determine the current 
mode of operation of said device. 

10. A device as claimed in claim 3, wherein control means are provided to control the device to provide an associated 
function when a predetermined orientation and/or movement is recognised. 

1 1 . A device as claimed in claim 1 0, wherein said device is usable only if a predetermined orientation and/or movement 
is recognised. 

12. A device as claimed in claim 1 0, wherein said associated function is a function of a user interface of said device. 

13. A device as claimed in claim 2, wherein said means for detecting the movement of the device comprises at least 
one accelerometer. 

14. A device as claimed in claim 1 , wherein said means for determining the orientation of the device comprises at least 
one accelerometer. 

15. A device as claimed in claim 13, wherein three accelerometers are provided which are arranged to provide infor- 
mation in respect of three orthogonal directions. 

16. A device as claimed in claim 1 , wherein said trained signal processing means comprises at least one neural net- 
work. 

17. A device as claimed in claim 16, wherein said trained signal processing means comprises at least one neural 
network is provided for recognising the orientation of said device. 

18. A device as claimed in claim 1 7, wherein the at least one neural network uses a self organising mapping technique. 

19. A device as claimed in claim 3, wherein said trained signal processing means comprises at least one pattern 
recognition system. 

20. A device as claimed in claim 1 9, wherein the at least one pattern recognition system uses a hidden Markov model. 

21. A device as claimed in claim 1 , wherein the device is a mobile telephone. 

22. A device as claimed in claim 3, wherein the predetermined orientation and/or movement of the device is pre- 
programmed. 

23. A device as claimed in claim 3, wherein the device is arranged to recognise the predetermined orientation and/or 
movement of the device which is selected by the user. 

24. A method for controlling a device comprising the steps of: 

determining the orientation and/or movement of the device; 

using a trained signal processing method to process the result of the determining step; and 
controlling the device in response to the determined orientation and/or movement of the device. 

25. A method as claimed in claim 24, wherein in the processing step, at least one predetermined orientation and/or 
movement of the device is recognised. 
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26. A method as claimed in claim 25, wherein in the control step, an associated function is provided when a predeter- 
mined orientation and/or movement is recognised. 

27. A method as claimed in claim 24, wherein said trained signal processing method uses at least one neural network. 

28. A method as claimed in claim 24, wherein said trained signal processing method uses at least one pattern recog- 
nition system. 

29. A mobile communications device comprising means for determining the orientation of the device. 

30. A mobile communications device comprising means for detecting movement of the device. 



55 



11 



EP1 104 143 A2 




EP 1 104 143 A2 




13 



EP 1 104 143 A2 




14 



j 



(19) 



J 



Europalsches Patentamt 
European Patent Office 
Office europeen des brevets 



(12) 



(88) Date of publication A3: 

01.09.2004 Bulletin 2004/36 

(43) Date of publication A2: 

30.05.2001 Bulletin 2001/22 

(21) Application number: 00309610.4 

(22) Date of filing: 31.10.2000 



(ID EP1 104143 A3 

EUROPEAN PATENT APPLICATION 

(51) mt CI7: H04M 1/00, G01C 21/20 



(84) Designated Contracting States: 


• LIukkonen-Olmiala, Tea 


AT BE CH CY DE DK ES Fl FR GB GR IE IT U LU 


90940 Jaali (Fl) 


MCNLPTSE 


• Hynnlnen, Tiina 


Designated Extension States: 


90500 Oulu (Fl) 


AL LT LV MK RO SI 


• Pirkola, Jani 




58212 Linkoping (SE) 


(30) Priority: 29.11.1999 GB 9928182 


• Sippola, Leena 




90500 Oulu (Fl) 


(71) Applicant: Nokia Corporation 


• Mantyjarvi, Jani 


02150 Espoo (Fl) 


90230 Oulu (Fl) 




• MMntyla, Vesa-Matti 


(72) Inventors: 


Fin-90500 Oulu (Fl) 


• Kaartinen, Sanna 


• Tuulari, Esa 


00140 Helsinki (Fl) 


Fin-90440 Kempele (Fl) 


• Kinnunen, Timo 


• Seppanen, Tapio 


92430 Paavola (Fl) 


Fin-90550Oulu (Fl) 


• Lustila, Risto 




96400 Rovaniemi (Fl) 


(74) Representative: Style, Kelda Camilla Karen et al 


♦ Salomaki, Lauri 


Page White & Farrer, 


96100 Rovaniemi (Fl) 


54 Doughty Street 




London WC1N2LS (GB) 



CO 

< 

CO 

o 



Q. 

LU 



(54) Handheld devices 

(57) A mobile communications device comprising: means for determining the orientation of said device; and trained 
signal processing means for processing an output of the determining means. 



-15 



10^ 
12_ 
14 



24 



A/D 




18 


16 




LPF 












HPF 






20 









Calibration 



X 



22 



A/D 



V 26 FIG, 2 




Printed by Jouve, 75001 PARIS (FR) 



EP1 104 143 A3 



European Patent 
Office 



EUROPEAN SEARCH REPORT 



Application Number 

EP 0O 30 9610 



DOCUMENTS CONSIDERED TO BE RELEVANT 



Category 



Citation of document with indication, where appropriate, 
of relevant passages 



Relevant 
to claim 



CLASSIFICATION OF THE 
APPLICATION {lnt.C1.7) 



X,P 



WO 99/22338 A (WILLIAMS HILARY LYNDSAY ; 
BRITISH TELECOMM (GB)) 
6 May 1999 (1999-05-06) 

* page 6, line 1 - page 7, line 2 * 

* page 9, line 1 - page 18, line 27; 
figures 1-16 * 

WO 00/63874 A (STRINGER JOHN WARREN) 
26 October 2000 (2000-10-26) 

* page 3, line 13 - page 6, line 15; 
figures 1-14 * 

* page 9, line 25 - page 21, line 1 * 

EP 1 104 143 A (NOKIA MOBILE PHONES LTD) 
30 May 2001 (2001-05-30) 

* the whole document * 



EP 0 833 537 A (MATSUSHITA ELECTRIC IND CO 
LTD) 1 April 1998 (1998-04-01) 
* the whole document * 



The present search report has been drawn up for all claims 



1-15, 

21-27, 

29,30 



1-15, 

21-27, 

29,30 



1-30 



1-30 



H04N1/00 
GO1C21/20 



TECHNICAL FIELDS 
SEARCHED (lnl.CI.7) 



G01C 
H04M 



Munich 



Dale of completion of the eearch 

9 July 2004 



Examiner 

Fourrichon, P 



CATEGORY OF CiTED DOCUMENTS 

X : particularly relevant if taken alone 

Y : particularly relevant if combined with another 

document of the same oategory 
A '. technological background 
O : non-written disclosure 
P : intermediate document 



T : theory or principle underlying the invention 
E : earlier patent document, but published on, or 

after the filing date 
D : document cited in the application 
L : document cited for other reasons 



& : member of the same patent family, corresponding 
dooument 



EP1 104 143 A3 



ANNEX TO THE EUROPEAN SEARCH REPORT 
ON EUROPEAN PATENT APPLICATION NO. 



EP 00 30 9610 



This annex lists the patent family members relating to the patent documents cited in the above-mentioned European search report. 
The members are as contained in the European Patent Office EDP file on 

The European Patent Office is in no way liable for these particulars which are merely given for the purpose of information. 

09-07-2004 



Patent document 




Publication 




Patent family 


Publication 


oited in search report 




date 




members) 


date 


WO 9922338 


A 


06-05-1999 


AU 


770921 B2 


11-03-2004 








AU 


9358598 A 


17-05-1999 








CA 


2307780 Al 


06-05-1999 








CN 


1278348 T 


27-12-2000 








DE 


69824311 01 


08-07-2004 








EP 


1032919 Al 


06-09-2000 








WO 


9922338 Al 


06-05-1999 


WO 0063874 


A 


26-10-2000 


AU 


4473000 A 


02-11-2000 








EP 


1250698 A2 


23-10-2002 








WO 


0063874 Al 


26-10-2000 


EP 1104143 


A 


30-05-2001 


GB 


2358108 A 


11-07-2001 








EP 


1104143 A2 


30-05-2001 


EP 0833537 


A 


01-04-1998 


JP 


3111909 B2 


27-11-2000 








JP 


10107718 A 


24-04-1998 








CN 


1179073 A ,B 


15-04-1998 








DE 


69714367 Dl 


05-09-2002 








DE 


69714367 T2 


27-03-2003 








EP 


0833537 A2 


01-04-1998 








US 


5953677 A 


14-09-1999 



Sj For more details about this annex : see Official Journal of the European Patent Office, No. 1 2/82 



3 



This Page is Inserted by IFW Indexing and Scanning 
Operations and is not part of the Official Record 

BEST AVAILABLE IMAGES 

Defective images within this document are accurate representations of the original 
documents submitted by the applicant. 

Defects in the images include but are not limited to the items checked: 

□ BLACK BORDERS 

□ IMAGE CUT OFF AT TOP, BOTTOM OR SIDES 
e/faded TEXT OR DRAWING 

□ blurred or illegible text or drawing 

□ skewed/slanted images 

□ color or black and white photographs 

□ gray scale documents 

s/lines or marks on original document 

□ reference(s) or exhibit(s) submitted are poor quality 

□ OTHER: 

IMAGES ARE BEST AVAILABLE COPY. 
As rescanning these documents will not correct the image 
problems checked, please do not report these problems to 
the IFW Image Problem Mailbox. 



